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P^ge 1 

..Models are convenieh^t expressions of How wt think things ought' 
to be. . When we use models to understand experience, however, it 
is inevitable that less than a per'f ett' explanr^t ion results/ This 
need not mean, the end •of the model » By *Stud;^ing the residuals 
from our expectations we can learn from, and come to, a better 



understanding of our experiences. 

• • -* " 

The impQrtance of inspecting residuals to-^gain information , 

leading to better niodels%?nd better understood data has been 

clearlj/ deftioijjstrated in analysis of var.iance literature by, 

among others, Anscombe and Tukey (1963) and Draper and Smith 
* * 
(1966) • Likewise, the analysis of covarianC'e structures depends 

on the 'interpretion of 'residuals in the model testing 

stage,s{Joreskog,19?9) . The .psychometr ic li terature, however, 

does not reveal as comprehensive an effort at the investigation 

of: residual patterns in order to un'derstand when and 'why data 

devia^te from a model. For example, vwhen the' Rasch model is 

applied to dibhdtomous data^ the bast analyses ^nay compute fit" 

statistics for items across people end for people across items 

"(Mead^ 1975), ' But these st:ati/ti,cs are o^ten' inadequate to 

locate the source of modtel departure. 'And, since these statis- 

tics, are sensHive to sam^ple size and^t^st length there is con- 

troversy regarding the jnagnitude^to be regarded as misfiJrtfnq. 

, ■ ■•- • , / ■ _ 

Although these summary statistics have, demonstrated their use* 

***^ 

f ulnfe'ss^'the^analysis of modeX fit must be carried further^ Just 

'> . ' * . 

as Draper and Smith oirge the display of residual patterns 
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addition to summary stati sties * J/ so may the^samie* approach;be 
* ^ • 

applied in our work. This paper illustrates some rt^chnigues 
* * . ■ «. 

found useful for exploring residual patterns resulting from the 



difference between observed and e'xpected scores as predicted by a 
Rascb model for polychotomous \aata developed by -Geoff 'Mastets 
, (Masters, 1980) Masters and Wright, 19B1). * 

We can derive three alternative forms with whjich to compute a 
■ * • 

residual from the model. The observed minus expected sco|^e yields 
what we call a score residual, Th*e score residual divided by its 



variance is call-ed a logit residual. The score residual divided 




by Its standard deviation i-s called a standardized residual. 

Our choice of the residual to inspect was based on hoH^ell it 
provided useful information. The ^core residual was rejected 

because floor and ceiling effects restricted the variation in the 

» * ' ♦ • , y 

extremes. ^ This led to our formulation of the logit residljal 

which proved to be sensitK'e to \^ariations in* the extrem^!a^.. but 

perhaps too much so. The same patterns could be found when com- 

paring logit and standardized< results ^ but the adjustment to 

tables and pictures in order to handle the magnitude of logit 

residuals proved to b€**>n:^Trh^onyenience. Since tile standardized 

residuals manifest 'a familiar metrdc, interpretatioji of their . ^ 

patterns becanfe the easiest of the choices* ' . . ' 

* . . ' • . , ■ 'a 

In Sloane*s (1981) disc\jssi^ the summary .approach 'to residu- 
als Suggestec) some misf it^in^ itenTs* Positive fits' cesult ^hen , 



more able children score lover than expected (yielding negative 

/ V ' ... 

residuals) and less able children score higher tl)an« expected 

(yielding positive residuals). N,egative fits result whe'n able 

chilSten do better than expected (positiv/e residuals) and less 
* • * ' , *. */ * 

able children do worse -than expected (negative residuals). But 
these summary statistics do not tell us exactly who has ^scored, 
unusually on which itemsr nor how widespread the problem is^ 

rThe first picture created to study this H<?as a plot ot statid- 
^ • ' • ^ J 

ardized residuals against, child abilities (Figure 1). This plot 

contains the original sample of 500 children. From^left to right 

the abilities increase. X' s'-represent 10 or more clVjldren with a 

given ability a^nd residual.. Each col\imn of points shows the. 

spread of residuals for the children with that ability^. For 

instance,^ the leftmost column shows the ' 14 residuals for the 

least able child. The picture we expect for data fitting the 

moc^el is a randoQ^ pattern with a m^n near ze.ro and a standard 

d&viation near one for each vertical array of residuals. ' ^ 

- ''What we see in. Figure 1 indicates something ^Ise. A rectangle 

marking off plushnd minus 3 standard deviations was added to the 

picture to highlight the asymmetry of the distribution in the 4.th 
» 

quadrant. ^ The points outside the^ ar ea are _ j:Aildi:„ej[i_ with 

^ — - ^ - - . ' " , * 

large negative residuals. It is recognized that very tble chil- 

dren will appear unusual only when they miss easy items, henpe 

Marge negative re'^idoals. Likewise, the least abld children will 

only appear unusual when they Succeed on relatively hard iAems, 
9 \ 
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yielding large positive residuals. The task, then, becomes one 
'of f indingi out* who ahe able children are, which easy 'items were 
"missed, and whether a valid explanation, other than one of chence 
.occurence, can be suggested for the large negative refeiduals. 

. ..One way see what is happening is to print 'the person by 
item residual matrix and to scan 'it, for patterns. This matrix, 
,can be constructed according to various sorting schemes depending 
upon what aspect the investigator wishes to highlight. It maj^ 
, Mso be i>uilt according tfo group membership. ^SucTi, a matrix may 
indicate unusual behaviors individuals or,grout)S of people have 
had on single items or blocks of items. Examples of this type of 
' matrix can be found in Wright' and Stone (1979). 

Figure 2 shows'- such a matrix. In this example the columns ape 

items sorted by their difficulty. and the rows are children sorted 

by their ability^ * Both sorts are descending. Various summaries 

may be collected in the margins. The entries in the matrix are 

standardized truncated residuals. Any residual with an absplute 

- - - ^ * 

.value equar to Ox greater than 9 was set equal' to 9. The summary 

'statistics, however, were computed from real, valued Standardized 

residuals. In these matrices, a criterion must^ be set so that 

data ace sufficiently focused whpn person by item interactions 

^are brought out. This allows for the matrix t# be' simple enough 

.to gVasp; This criterion was set *such 'that only negative residu- 

als equal ^o or less than^-3.0 and only children who had at least 

onfe such re^idijal ate displayed. This matrix concentrates on 
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surprising mistakes* 



In the SUBJ. ID .field in -the left margin of the matrix, the 
first 2 columns are the age of the child in mcJnthS and the third 
column is the child's sex.' 0"L". fof boys', '*2" for girls). The 
t/ight most margin "has* the child's logit ability. 

Inspection of the ID field suggests that boys and 'girls, ^re 

sp^read equally throughout the ability" range. There does appear 

• 

to be some age differentiation, however. The older children gen-- 
erally .have the greater? abilities .virile the younger chilfilren are 
-primarily in the lower ability range. The fact that older chil- 
dren perform better'^Dn a aevfilopm^ntal instrument is hardly sur- 
prising* Of interest to us is whether the failures relatively 

easy items can b^ explained by a characteristic of .the item ihat 

i • • • ■ ^ ■ 

le^^o a related group o^ . children ha^^ing had- ui)e<^ected trouble 

with it. - * , - , ^ 

/ • » # ft 

f\ m ^ 

In order to simplify ^his demonstr^^tion we chos^ to emphasize 
what^ appeared to be/' an age factor operating In. an unexpected 

j m ■ 

direction. First ye/controlled for sex by looking -only at boys 
anj3 then selected jhe 56 oldest^ and> 42^youngest ones. • 

• * * ■ 

Figure 3 is the pl6t of the 98 boy abilities by their- res/du- 

f # • 

Sis. This plot Correst)onds with that of. Figure 1. At this point, 
though, Ve do not know the age 'of the most able% chi Idren in Pig- 
Ujre 3, nor what, items are involv.ed, nor if there 'is even a dif- 
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ferencQ in the abilities distributed among the .two age groups. ^ 

I 9 m 

Figure 4 present's the residual matrix for the reduced, s'et of 

98 boys. In the ID field the first 2 columns are the- age and 

the third column is the age grc^up classification .(^1^ it the ag^ 

is less tha^ 47 moaths, ^2^ if the age is greater than 59 

months). The criterion wds set for negative residuals equal to 
t * ■ 

or less than -2.0. , This matrix highlights those more able chiL- 

[ * * • * 

dren who- have done worse than expected • 

, ■ •. . ■ ■ • 

This concludes our discussion of search techrviques for under-_ 
. >. — ' " / ' 

standing the summary statistics used by Sloane. *These results do 

not completely explain her misfitting items because the girl^ are 
not included as part of, our older and younger boy dichotomy. The 
point should be cle^r, however, that the' misfit in Sloar^e's ori- 
ginal so}.ution can be explored through a residual analysis. 
Inspection of th^ residual mat^ri'ces is one way of understanding 
why an item has misfit the models 



Now we look ajt '^e subtsam'feie of '98 boys to see if the varia- 
ble definition has remained the same for the two age groups'. Our^ 
Intent is to. demonettfate a process* that may be applied. whenever/ 
the question is ask^d *'Have these groups of people petformepi th 
same on the Instrument?** 

Figirre 5 'is an ability frequency distribution iftap. The young- 
est boys are on the upper map, the older boys on the lower/ .The 
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numbers indigate the number of people located at the same posi- 
tion. Double digit numbers are read vert,icaljy, *e»g. . in ^the sec- 
ond map the 1 above the, 8 refers to 18 peo'ple with an ability 
near 3,0, ' ^he M and S refer to the respective meahs and standard 
tiievi'ations . Figure' 5 shows how much more able the qMer boys ar^e 
than the younger. Figures 6 and 7 show the ability oy'resiidual 
plots for the younger and older boys, respectively* There tA^ere 
no negative abilities for the older boys so we^ revioved the nega- ' 
tive side of th'e plot fromTigure ?• , A comparison with Figure 3 
confirms that tlie large negative residuals are due. primarily t3 
the oldeV boys,. We npw turn to the investigation of the items. ^ 

Figure 8 is a' plot of residuals again,st item difficulties. 
The items range from easiest at the left to hardest at the right.' 

T 

^The rectai>gle marks off plus and minus % standard deviations. 
The 3rd quadrant shows that the easier items have the Idrge neg?- 
tiye residuals. This picture conf ims 'what we learned from Figures 
3* and 4. . . . ^ * 

The information in the sorted matrices of Figure's 2 and 4 is 
convenient when the sample in the matrix does not exceed 100. At 
_ tnat point three pages are .required to present the picture. This 
motivates us to seek another way "of presenting the- residual 
ihfocmation for large samples or when group comp^rispns' are to be 
ma^e. A useful way to compat^e groups in terms of. residual^ dis- 
tributions over individual • items is shown in Figures 9 ^nd lOi 
In Figure 9, for ITEMS, we see the residya} .distribution for each 
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group of boys on a line extending from -5 to +5 standardized 
resi'du£fl units. The integers on the line represent the number of 
beys who had a gi\^^en residual value. Again the numbers are read 
vertically and the^ M and S represent the group means and standards 
deyiationa "Of these .residuals. 



This type' of map' allows a detailed examination of within and 

r * 

between group performances on individual Items. Also, we can 
combine items that kx^ similar and bu;ld these maps fo.r groups of 
people across items. Not .only can find wherje surprising 
behavior has occurred but we can identify which children and 
>^tems'^re involved. 

A useful summary of jall 'the i tem maps is contained in Table 1. 



The Table contains 



the group means and standard deviations for 



3 a 

each it^m*s residuals*.' This table may be used in a variety of 
wa/s. 'We could talce the difference between^ group means and look 
at the items with the larges.t differences. We could look at each 
item where there wa? a difference in the sign of the mean. For 
this example; we chQse: tp CQOcentrate on the 2qd columns of means 
and ^tandari^deviat ions, in particular, the negative means. 



When data fit the model, residuals are ability and difficulty 

free and differences in summary statistics should be attributable 

to chance. We have seen in these data, however,' a combin action o£ 

< ' • . ' . . 

high ability and low ' "dif-f iculty that results in high negative 

C'esiduals. S.ince^the older boys' are generally the more able we 
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expect them to h'a^i^ higher means and s'hialler "standard devjiatidns 

in the re^siduals relative to the younger ones. The negative 

means for tb$~^der boys under column, 2 represent instancy wherfe 

the mean performance by the older boys was 'worse in terms of 

^. ^ ^ \ • ^ ' t 

expected behavior than that expected . from the younger ones. 

Sirjce ^his instrument vas designed to meaq^re development it is 

surppisi^ng^o see some of the more able, older boys ]&er forming 

con^derably worse than expected. This is not to say their per- 

^ », 

,formanc*e was bapl-^ just that it was not as good as was expected. 

♦ 

« 4 

Me need^o knpw if the surprising residual means can be 
explained ^by chance or whether something more fiindamental is at 
stake. 'From inspecting, the maps for ITEMS anfi ITEM14, in figure 
9, i\ is evident that outliers (indicated by , asterisks) have. 

m 

Skewed the means of the-older boys. These children may aiso be;J 
found in Figure 4 under the two items in question. After tracing^ 
the problem down to two outlifers oh each ijtem, %tfe conclude the 
items are functioning as intended but we could take the matter 



furtljer if we cared to becaiise we have identif ied. specif ic chil- 
dren" who had surprising trpuble. , ■ * . 4 



Looking at -the residual distributions for- ITtM6, ITEM? and 
rTEMll, in i?igure-.10, we see that in each case the oldet boy 
distribution is bijnodal with a group of children about one^ stan- 
dard deviation below the mean. Those witlvpositiye 'residuals arfe 
not surprising. ' Those with the negative* residuals suggest a pat- 
t^rn of deficiency or erroneous observation and we may ask if .tHe 
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larger, negative residuals are from the same chilctren. From a 
matrix of residuals wi^th a criterion set at *1 we found that, in 
fact^ .%ome of the sam^ children j^ad large residuals on combina- 
tions of two of the three items and these children were in the* 
middle range of the ability distribution for the older boys. 
'Apparently the problem with ITEM6 and ITEM? iS due to a 'few of 
the same older boys. Since the median for these items is cpnsid- 

r ^ • ^ 

erably to the right of the respective means we suggest these 
^ items are functioning as intended* It would be 'teasonable , how- 

• - ' •• • 

ev€r, to monitor these items in future applications. 

Such an interpretation is not possible, howev.er, for ITEMII in 
Figure 10* Here the median approaches the mean but the, distribu- 
tion is ^still bimodal. Whp made the mistakes on this item? 
Interestingly, it was the most able of the older boys* i*e detep^ 
mine this by looking ageiin at Figure 4 and^. noticing the largest 
residuals beloftg to boys in the, highest ability lev^.^Jhe prob- 
'lem^with ITEMll may be due to' confusion on^ the part of some Of 
the older boys who did |/ot know to whonT^he *'in front of 
instruction, was diVected* 



Where does this leave us? We- could be a^keS why we did not 
calibrate the i terns. ^^parately for the two ^groups of boys, and 
^ plot the item difficulties against one another with error b'ands^ 
This wars done in Figure 11.^0 underscore why it is not necessary 
and can be misleading* One Reason for explofing the"^ rjes^duals 
f^rom a total sample calibratign .was to show that the game group 
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*» ' ' ' ^age 11 

• i ■ * * ^ * 

differences/are^ seen as when separate caTlbrations are done.. The 
fir^t thing to check in 'Figure 11 is whether the saire item^ th5t 
the residual^ analysis tagged as\'peculaar also stai>d out in the, 
plot.^ We .See they, do in fact eitj^er lie outside^or close to the 
3 standard error baads* .important pdinlil^ however, is not th^ 

jcortoborat ion* . Jf we saw, only this plot we might conclude there 
was a variable definition problem in tl\pse data, one where the 

items m^eant on^ thing to th^ older , boys and anotlier to the youn- 

■ ■ ' ^" * " ^ ' * ^ ' o 

get^ But fhis is .not the- cas? as we have ' Seen by exploring and 

^xplain-ing •the're.siduai patterns," Th^- P,lQf in. Figure 11 merely 

' - ' ' ^ jft / 

indicates t'hat 9ertajln items aret p«uliar but doe^^bt explain 

why*^; The same argument holds true if one" is interested in com-, 
puting^ t^statistcs to test the difference bjetween paits of item' 
dilf idiltieSf The residual analysis not only identifi.es the. same, 
itejps blit" tnay^ explain w;th judicious invest^igat ion why^there was 
a peculiarly and shows that, €txcept for' ITEM^l^it is not neces- 
sarily" a problenj^ of 'it>em coniJtruction. / . — 



In* conclusion, we arque that a systematic analysis of residu- 

als offers the investigator a de^cision facilitation technique not 

found in the conventional summary fit sfa-t^stics* The process 

may be used to understand individual people, individual items, 

groups of |)eople or groups of items. Such an understanding of 

* * r ' • 

resTdual patterns may prove' useful as/ a mqans of addressing 

I'ssues of 'item bias* '^'quessing*^ and ^discrimination'. 
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